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Abstract 

Requiring savvy, online courses can present a perceived barrier to adult 
learners for a variety of reasons, but ease of access due to inadequate 
navigability should not be one of those barriers. Re-tooling online courses to 
standards of Quality Matters (QM) can lessen that barrier to access, making 
adult learners more participatory. Internal statistics gathered routinely by a 
learning management system (LMS) can test this claim, as a way to show 
successful, measurable increases in participation. Though only an 
approximate method at this time, as presented herein, a detailed examination 
of user-progress data from a standard, stable LMS appears nominally 
promising for development into a prospective analytic tool of assessment of 
adult learners. 
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A typical learning managements system (LMS), like Desire2Learn (D2L), 
collects an ongoing amount of data about students as they access and 
navigate a digital environment for a given online course (Brightspace 
Tutorials, 2015). These data, known as user-progress data, are readily 
available to the course's instructor in an effort to track performance, identify 
problem areas, and T to maintain an accurate assessment of levels of 
participation as a basis for grading. At National Louis University, the use of 
these data for tracking students is similar (Brightspace Tutorials , 2015). 
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Once the course is completed, these data are still available for instructors to 
return for a deeper assessment of whether the course was cleanly accessible 
to students, oiy as one means to determine whether there are unintentional 
roadblocks and barriers that prevent students, especially adult learners, from 
accessing online content. In that assessment, one can likely turn to ways to 
make content more readily accessible for adult learners. For example, in 
recent years one way requires the application of principles utilized by Quality 
Matters (Underlying Principles of Quality Matters, n. d.). 

Quality Matters (QM) is a nationally recognized and tested system of 
increasing the utility of online courses by streamlining these courses' 
navigability for potential and active users. Colleges and universities that 
apply QM principles to their online courses find greater accessibility due to 
greater ease of online navigation within those courses. That is, prior research 
on the efficacy of Quality Matters has shown its demonstrable improvement 
for the navigability of online courses (Regon, 2007). It is the intent herein to 
present a simple, straightforward approach by which one might measure that 
improvement in navigability using data gathered by a LMS. 

Re-tooling a course in QM, after evaluation by a team of qualified QM peer 
reviewers, can allow a course's content to become more readily accessible. 
Though QM makes no determination on the quality or quantity of a course's 
content, its application to a course does allow an instructor to make the 
course's navigation cleaner so that access to its content can be improved. 

The initial premise of this paper is that user-progress data, gathered by a 
LMS, can be used as one means to determine whether a course can become 
more readily accessible after an application of QM principles. A supporting 
premise is that online courses, once defined by, or revamped under, QM 
principles can allow greater access to online users, such as adult learners, 
than they might otherwise have an opportunity to navigate successfully. 

The overall assessment from these two premises is based on the prospect of a 
relatively measurable rate of improvement through a comparison of user- 
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progress data of two courses gathered by a LMS, namely, D2L, through 
which courses, before-and-after QM re-tooling, were offered at National 
Louis University. 

Since the user-progress data were readily available from two NLU courses, 
both before and after a controlled application of QM principles, it seemed a 
likely test of whether QM re-tooling can be as efficacious as QM asserts. To 
do this effectively, under controlled circumstances, one would need before- 
and-after data that can be compared from an offering of the same course at 
different intervals. Ideally, it would be useful to have an identified cohort of 
students on which to compare these data, though that may not be possible at 
all times. Also, it would be necessary to have some indication that the course 
was composed entirely of adult learners, in order to test the validity of the 
two premises. 

Note that while the number of students from course-to-course ought to be 
more equivalent in number for an accurate comparison, there is no actual 
control for the number of students signing up for any given course. To set the 
premise for any comparison, though, one can assume that the majority of 
students in any given course at NLU tend toward the expected demographic 
of an adult learneiy since that is the anticipated audience to which NLU has 
appealed for students. 

That is, these students are often, but are not limited to, those who are greater 
in age than the typical career student, working at one or several jobs, and, 
have genuine, day-to-day family concerns. This demographic often includes 
students returning to complete an academic degree, either undergraduate or 
graduate (NLU at a glance, 2014). Pertinent to this sample, it was fortunate to 
be able to compare two science courses, primarily in an effort to determine 
the efficacy of QM as one more means to assist adult learners in 
apprehending the content increasingly provided by universities in an online 
environment. As QM does require a certain level of online familiarity for 
users to get the most benefit, it seemed reasonable that one may be able to 
track the efficacy of a website's improved navigability through the use of a 
LMS. While a lack of familiarity with digital technology may be problematic 
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to certain student populations, it can be perceived as a distinct mental barrier 
to participation for adult learners. QM may offer the chance to detect and 
reduce this perceived barrier. 

In addition, as adult learners, students in an online environment may confide 
a lack of ability to their instructor in an attempt to preface their performance 
within a digital environment if they have not had productive exposure to 
online courses. QM also appears to present an opportunity to encourage those 
same students to convey a greater level of confidence in their ability to 
apprehend content better with greater ease of navigation in an online 
environment. 

Conceptualization 

The technical requirements to access a typical learning management system 
may discourage adult learners from engaging in an online course. This can 
present a very real barrier to successful progress, requiring intervention by 
the host institution. Though adult learners are largely self-directed in taking 
online courses, it is a trait of adult learners that if they continually find the 
course inconsistent in its presentation or difficult in its navigation, one can 
reasonably assume that adult learners will participate less in the course as a 
whole (Pappas, 2013). As a systematic means to improve course presentation 
and navigability, QM can encourage greater participation for adult learners. 

While QM's utility is demonstrable among online courses, its success as a 
way to reduce the perceived barrier that online courses presents to adult 
learners can be shown, but with obvious and prudent caveats. That is, while 
QM does improve the navigability of online courses, its specific 
improvement among adult learners can be best measured under controlled 
circumstances, allowing for measures that take into account specific 
requirements of adult learners. To that end, this presentation offers a small 
measure of possible evidence to bolster this latter assertion. Comparing the 
same kind of online courses that do (and do not) employ recognized 
standards of QM appears an adequate way to test whether adult learners can 
increase their rates of participation to access online courses. That is, one can 
test for increased rates of participation before and after the application of 
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QM, thereby drawing a measurable inference, not only of the relative efficacy 
of QM, but as a means to lessen the perceived barrier. 

Design & Development 

QM offers a tested means to organize a cleanly accessible online presentation 
with straightforward navigation to encourage greater participation. While QM 
does not validate or assess the specific content presented in a given online 
course, it does offer the instructor helpful recommendations to organize that 
content, promote its presentation, and streamline its access (Linkedln 
SlideShare, 2014). As application of QM principles can improve students' 
online experience, it offers adult learners confidence in online navigation. 
Greater confidence can lead to greater participation. To test for this greater 
participation, one can readily compare science courses before-and-after QM 
re-tooling. General education science courses offer a good environment, as 
they often have little subjective content. This makes the tally of user progress 
unambiguous. Fortunately, a LMS does often employ internal statistics to 
monitor the progress of students through specific online courses. This user 
progress is often typically available to the instructor as one of several online 
analytic tools (Meyer, 2015). 

By comparing levels of user progress for the same courses for which specific 
content is presented, one may determine a relative rate of access by adult 
learners before-and-after re-tooling. The rate of access can be gauged by 
activity levels of adult learners during randomly chosen times during which 
all students are obliged to access specific content. Internal statistics on user 
progress for any chosen time frame typically reveal amounts of time spent on 
areas within the LMS for that course. 

To attempt this comparison, an existing 300-level natural-science course was 
chosen that was representative of adult learners attempting to take and access 
an online course. The course, LAN 300 Ecology, is an online course with a 
laboratory component, according to the NLU Course Catalog (NLU Course 
Catalog, 2016). There are two main reasons for its straightforward selection, 
namely, the course was a relatively long-standing course in the roster of 
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courses offered online at NLU, but it was offered as an online course on a 
well-known and stable LMS; and, though its content and textbook were 
largely the same, QM principles had been applied to the layout of the course 
over the past year (i.e., by 2015) to make it more straightforward to navigate. 

In selecting this course for its inclusion in this presentation, there are several 
obvious caveats. For example, for this to be a truly accurate assessment of 
course navigability before-and-after an application of QM principles, the 
class would have to consist of the same composition, size in numbers, and 
age range of adult learners. Ideally, one might need the exact same class, in 
order to assess improvement from learners “before” QM to learners “after” 
QM. Of course, no class would likely take the same course twice. 

In addition, to show conclusive results of a demonstrable improvement for 
adult learners, that is, for a true course-to-course comparison of progress 
before-and-after QM re-tooling, the course content would need to remain the 
same (or largely the same, after updating, as needed) while its navigability 
would be streamlined by application of QM principles. In this latter instance, 
that course content was preserved. So, though the comparison is not learner- 
to-leamer, an initial comparison can be made course-to-course. 

To a lesser extent, then, it would be useful to know the identity (i.e., relative 
age) of those adult learners taking this course successively online. While 
personal information on students is not made available for this presentation, 
one can infer from its weekly discussion forums that a majority (if not all) 
were adult learners. This inference can be gleaned from the incidental content 
of the discussion forums, namely, by what the students revealed about 
themselves as they took the 10-week course and attempted to meet its online 
requirements of assignments, quizzes, discussions, and a final exam. 

Implementation 

The straightforward approach of applying QM has already been underway at 
NLU since 2014 (Donahue, 2014). This is why it can be instructive now to 
compare selected courses before-and-after QM re-tooling. To determine 
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whether the application of QM principles to streamline the navigability of 
online courses might actually improve their accessibility for adult learners, a 
pair of natural-science courses, made available online one year apart, were 
selected for an examination of their user-progress data as archived by the 
LMS on which the courses were offered. That is, two courses of LAN 300 
Ecology were selected as representative of before-and-after examples of QM 
re-tooling. The first ten-week course was offered in NLU's Spring Quarter 
2014 (from 04/01/2014 to 06/03/2014) and hosted eight students. The QM- 
retooled course was offered in Spring Quarter 2015 (from 04/06/2015 to 
06/14/2015) and hosted sixteen students. Re-tooling to QM occurred just 
prior to the start of the course in SQ 2015. 

To effect a comparison between these two courses (viz., LAN 300 Ecology 
offered at two distinct times) before-and-after QM re-tooling, user-progress 
data were compared from both courses. On the LMS known as D2L, user- 
progress data indicate the actual times of logins, and length-of-stay during 
those logins, among all enrolled students for the duration of the ten-week 
course. This collection of user-progress data is a built-in analytical tool of 
(Brightspace Tutorials, 2015) to assist teachers in tracking how often and for 
how long students are actively engaged in their assigned work for an online 
course. 

From the initial two premises of this paper, it is assumed that an established 
rate of logins over a distinct period of monitored time within a given course 
may offer an indication of the relative barrier that online courses might offer 
to adult learners. That is, if online courses present a true barrier to adult 
learners, that barrier may be perceived by the frequency, or lack thereof, of 
logins over a selected period of time as students become acclimated to a 
given online course. If the frequency of logins appears to increase, or, if a 
previous dearth of login frequency does not manifest itself after the 
application of QM principles, all other things equal to the extent that can be 
known, then a course re-tooled by QM may be perceived as less of a barrier 
to adult learners than a non-re-tooled course. Repeated logins and a relatively 
high percentage of completion of content may infer a constant search for 
content and a sincere level of engagement. By comparison, few logins and 
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scant time spent in any given area obviously show a distinct lack of 
engagement and interest. While the former instance may show a genuine 
attempt by adult learners to access content, the latter instance offers no other 
logical conclusion than disinterest as an adequate explanation. 

Of course, one cannot gauge access based on time of day, since online adult 
learners will access content asynchronously; rather, one likely has only to 
look at both the frequency of login and each login's duration to gain any 
insight into any hesitancy with accessing an online environment. So, without 
any other corroborating evidence, such as specific complaints from students 
about inaccessibility to content or difficulty in navigation, one can reasonably 
assume that poor presentation and difficult navigability present distinct 
barriers to the learner's participation. 

For a QM-tooled course, the increased number of consistent logins over a 
defined period of time and their increased duration can infer a relative ease 
with access and navigability. Coupled with a relatively high rate of successful 
content completion, as indicated by the LMS analytics, one can infer that QM 
offers improved presentation and navigability, which lowers the barrier to 
adult learners' participation. 

Evaluation 

A formal, side-by-side comparison of the same course, before-and-after QM 
application, may show similar, encouraging results to engage adult learners. 
This side-by-side comparison is worth exploring as host institutions seek to 
attract and retain adult student populations. Though the courses, as noted 
earlier, were not set up for review under controlled circumstances (that is, 
complete demographics of adult learners were not known with certainty and 
frequency of logins could not be directly attributed to a full application of 
QM principles), only a rough comparison can be made at this time. In a more 
complete evaluation, an adult student cohort participating in a series of 
before-and-after QM re-tooled courses with a built-in template to guide 
navigation and access would be needed for a more accurate assessment. 
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So, for the purpose of this paper, one can examine available user-progress 
data for a rough approximation. For example, comparisons of LMS user- 
progress data showed that the earlier course (in SQ 2014, pre-QM) had spotty 
access to content modules, and, logins appeared primarily to meet designated 
deadlines. Though the course offered a full set of content-rich weekly 
modules, the adult learners appeared not engaged. More to the point, the user- 
progress data, while useful to see logins over a period of time, show little 
regularity. (See Screenshot 1 .) Presumably, too, there were other reasons for 
poor engagement. For those learners who did access the previous course, the 
duration of time spent on content areas inferred that the content was 
presented in an unappealing manner. 

By contrast, the latter offering of that same course (in SQ 2015, post-QM) 
shows students far more engaged in accessing the course; that latter course 
has a slightly more welcoming presentation and clearer navigation, i.e., more 
neatly arranged content modules, than its previous installment as an online 
course. Like its previous installment, this latter course contained the same, 
full set of content-rich weekly modules, along with (at least) some QM-added 
navigational assistance (such as a welcome module). In the latter course, the 
user-progress data show more and regular access by all students. (See 
Screenshot 2.) Certainly, for example, twice as many students in the latter 
course would stimulate more logins, as students were required to respond to 
one another in weekly discussions, but the access is demonstrably greater 
than the previous (pre-QM) course. 

There are obvious caveats to keep in mind. Certainly one cannot infer too 
many reliable conclusions based on this rough comparison. This paper is 
intended simply to offer an insight for instructors who may wish to craft a 
similar assessment on their own, in order to determine the efficacy of QM re- 
tooling. This approach might be useful for those institutions that have started 
to apply QM principles, but requires more time to do a thorough job of it. As 
these institutions seek to apply QM principles to online courses, they might 
add to their QM re-tooled courses an internal (perhaps weekly) checklist 
(usually available on the LMS) to organize their adult learners toward 
completing various assignments, so that their user progress can be more 
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easily interpreted after the course concludes. While this paper offers just one 
approach of assessment, it is likely that a more controlled comparison 
between online courses, “before” and “after” QM, is necessary to have a 
more meaningful discussion of data. That controlled comparison may 
determine whether the application of QM principles can indeed improve 
navigability enough to remove barriers for adult learners to access course 
content. In the interim, it may be useful as well to seek user-progress data 
from the QM organization itself, in an attempt to carry forth this assessment. 

LAN 300, SQ 2014, pre-QM application 



Screenshot 1 . User-progress data, while useful to see logins over a period of 
time, show little regularity in access to 
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LAN 300, SQ 2015, post-QM application 
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Screenshot 2: User-progress data, while useful to see logins over a period of 
time, show demonstrably greater access in 


70 

ePublication © 2016. Copyrights are maintained by the authors of each chapter, www.ahea.org 


References 

Brightspace LMS with new analytics tools (2015). Retrieved from 

https://theiournal.com/articles/20 15/1 0/28/d21-updates-brightspace-lms- 

with-new-analytics-tools.aspx 

Brightspace Tutorials. (2015). Navigate D2L learning environment - user 
progress - user. Retrieved from 
https://www.voutube.com/watch?v=4cEZ6swmiw4 
Donahue, M. (2014). A new chapter for online learning at NLU [NLU 

blog post]. Retrieved from http://blog.nl.edu/2014/04/a-new-chapter- 
for-online-leaming-at-nlu/ 

Resource Center, (n. d.). Retrieved from 

https://documentation.desire21earn.com/en/User%20Progress 
National Louis University, (n. d.). [Reference Guide] . Retrieved from 

https://www.nl.edU/media/nlu/downloadable/lms/l 03/track student pro 

gress.pdf 

Linkedln SlideShare. (2014). QM's Providing Helpful Recommendations: 
What's the Formula? Retrieved from 
http:// www. slideshare . net/ mverm 1 /qms -providing-help ful- 

recommendations 

Meyer, L. (2015). (Brightspace tutorials, 2015) Updates. 

NLU at a glance (2014). Retrieved from 

http://www.nl.edu/about/nluataglance/ 

NLU Course Catalog. (2016). Retrieved from 

http ://nl. smartcatalo giq . com/ current/undergraduate-and- graduate- 

catalog/course-descriptions/lan-natural-science/300/lan-300 

Pappas, C. (2013). Adult learners' traits. Retrieved from 

http://elearningindustrv.com/8-important-characteristics-of-adult- 

learners 

Regon, R. University of Maryland online, 23 ld conference on distance 

Learning and Teaching. (2007). Research on the Impact of the Quality 
Matters Course Review Process. Retrieved from 
http://www.uwex.edu/disted/conference/Resource library/proceedings/ 

07 5284.pdf 


71 

ePublication © 2016. Copyrights are maintained by the authors of each chapter, www.ahea.org 


Underlying Principles of Quality Matters, (n. d.). Retrieved from 

https://www.qualitymatters.org/research-grants/fipse/principles 


72 

ePublication © 2016. Copyrights are maintained by the authors of each chapter, www.ahea.org 


